Kernel Conjugate Gradient is Universally Consistent

We study the statistical consistency of conjugate gradient applied to a bounded regression learning problem seen as an inverse problem defined in a reproducing kernel Hilbert space. This approach leads to an estimator that stands out of the well-known classical approaches, as it is not defined as the solution of a global cost minimization procedure over a fixed model nor is it a linear estimator. Instead, approximate solutions are constructed by projections onto a nested set of data-dependent subspaces. We study two empirical stopping rules that lead to universally consistent estimators provided the kernel is universal. As conjugate gradient is equivalent to Partial Least Squares, we therefore obtain consistency results for Kernel Partial Least Squares Regression.